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Introduction 

Are you overwhelmed by demands to be more "data-driven" in your practice? Has the latest agency 
management decreed yet another set of dashboards to guide agency decisions? Are you inundated by 
numbers that are impenetrable, make little sense given what you know, don't reconcile with the 
numbers you keep about your own caseload, or are about issues that seem only marginally related to 
what you do and what you can influence? 

Or maybe you are one of the lucky ones, working in a system of care with a well-developed, fact-based, 
decision-making culture? 

Whether frustrated by numbers or comfortable with using data in daily decision making, this practical 
guide will help you use data to determine whether the best outcomes for children and families are being 
achieved. 

What if you learned that your division had the highest average rate of out-of-home placement in your 
organization? Or that your teen peer-to-peer support program had the highest rate of psychiatric 
hospital reentry of all the peer-to-peer programs in your system of care? Wouldn't this kind of 
information provide an incentive to reevaluate your practice with colleagues and determine how better 
outcomes might be achieved? 

This guidebook unbundles the "data 
puzzle" and gives you important basics 
that will help you more effectively 
identify, use, and evaluate and monitor 
practices, services, and outcomes for 
children, youth, and families involved in a 
system of care. Data (both qualitative 
and quantitative) provides evidence, 
taking the emotion and guesswork out of 
what can be tough calls in making 
decisions. With an effective data-driven, 
decision-making system, states, tribes, 
counties, and communities can more 
easily analyze performance data by 
important subgroups, challenge 
assumptions, and address problems 
(Messelt, 2004). The facts become the 
key driver for quality improvement 
activities. 

This guidebook is intended to be useful 
to human service systems that support 
young people and their families across a 
comprehensive array of services within a 

system of care, including child welfare, mental health, juvenile justice, education, and health care 
(Figure 1). Each of these systems may have different goals and use data differently to determine child 
and family outcomes. 


Figure 1. System of Care Framework 


Child/Youth 

Development 

Services 

( Mental Health ' 
Services 

Child 

Welfare 

/ Juvenile 
i Justice 

Child 

and 

Family 

Educational 

Services 

1 Vocational 
\ Services 

F Substance 1 
1 Abuse I 

\ Services J 

Health 

Services 

Adapted from "A System of Care for Children and Youth With Severe Emotional 
Disturbances" (Rev. ed.), by B. A. Stroul and R. M. Friedman, Georgetown 
University Child Development Center, National Technical Assistance Center for 
Children's Mental Health, 1986. 


1 




Putting the Pieces Together: Guidebook for Fact-Based Decision Making to Improve Outcomes for Children and 
Families 


This practical guide focuses on the data 
necessary to build virtually every 
element of the infrastructure of a 
system of care (Figure 2). Promoting 
shared accountability for results across a 
system of care aligns the work of the 
system around common goals, helps set 
priorities, and instills pride in system 
accomplishments and outcomes 
(Hodges et al., 2007). 

Management by data, data-driven 
planning, or fact-based decision making 
describe the same concept: using the 
very best information to achieve the 
very best outcomes for children and 
families. In its most basic form, data- 
driven decision making is about 
collecting appropriate data, analyzing 
that data in a meaningful fashion, 
getting the data into the hands of the 
people who need it, using the data to 
increase program efficiencies, and 
communicating data-driven decisions to key stakeholders. In short, the facts will help justify directing 
limited resources to target populations or areas of service need and add credibility to your conclusions 
(Schrepf & Brown, 2011). 


This guidebook presents the four pieces 
of the data puzzle (Figure 3): 

1. The data itself: what you need 
and where to get it. 

2. The analysis of the data: common 
approaches to understanding the 
meaning of the data. 

3. The presentation of results of the 
analysis. 

4. The communication and 
dissemination of those results. 

This guidebook also includes a variety of 
real-life and hypothetical examples 
illustrating key points. In addition, as we 
explore each element, we follow the 
stories of three jurisdictions—Virginia, 



Figure 2. System of Care Management 



Adapted from "Building the Improving Child Welfare Outcomes Through 
Systems of Care: Building the Infrastructure: A Guide for Communities," by 
Children's Bureau, U.S. Department of Health and Human Services, 2007, p. 15. 
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New Jersey, and Erie County, New York—that have made, or are in the process of making, progress in 
improving outcomes by using data to inform a wide range of practice and policy changes. 


Virginia: Asking the Right Questions 

Governor Tim Kaine was elected governor of Virginia in 2005. Flis wife, Anne Holton, was a former juvenile court 
judge who made the needs of Virginia's children and families one of her primary advocacy issues during her tenure 
as the state's first lady. 

"The system was totally reactive," says Judge Holton. "We would wait until things were really awful at home, and 
then we'd remove the children, and maybe we'd find another family for them. Things would fall apart there, and 
we'd put them in a group home. They'd run away, so we'd lock them up in some institutional setting. In many 
cases these were really smart and interesting kids and the system clearly was failing them." (Walters, 2010, p. 4) 

Asking the right questions was the key to designing and sustaining a fact-based, decision-making culture across 
human service systems. Judge Holton knew from experience that the system had focused for far too long on 
process: what they did (e.g., number of cases investigated, number removed from home, length of stays in care) 
rather than on what they accomplished. 

That simple paradigm shift—from "what do we do?" to "what have we accomplished?"—began the process that 
fundamentally changed Virginia's approach to supporting families and using data to strengthen family- centered 
practice. 

New Jersey: Demonstrating Accountability to Stakeholders 

In 1999, Children's Rights, an advocacy organization, brought litigation against the New Jersey Division of Youth 
and Family Services (DYFS). In 2003, the suit resulted in a consent decree between the DYFS and Children's Rights. 
The decree contained a significant number of specific performance measures and goals—both for process and for 
outcomes—that would establish the basis for New Jersey data collection and analysis for many years to come. 

By 2006, although some significant progress had been made, all parties agreed to a modified consent decree that 
provided even greater room for collaboration and defined the need for a new practice model—as well as 
supporting data, data analysis, and management, and quality systems to support it. In New Jersey, critical 
indicators came from litigation and resulted in tracking and communicating data essential to demonstrating 
accountability that systemic reform has occurred and that outcomes are improving. 

Erie County, New York: Creating a System of Care 

Erie County has a population of approximately 920,000 and includes Buffalo—the second largest city in New York 
and the third largest poor city in America. Stakeholders in the community were concerned that there were simply 
inadequate resources for children at high risk of mental illness. Key-child serving agencies—including the 
departments of social services, juvenile justice, and mental health—collaborated with families to prepare an 
application to Center for Mental Health Services for funding to implement a system of care. The grant was 
successful and was awarded to Family Voices in 2004. 

From the beginning there was a commitment to the system of care values and guiding principles. Data across all of 
the systems informed fact-based decision making to reduce institutionalization of youth with serious mental 
health challenges and reinvest savings into community-based services and supports. From the beginning, they had 
to link financial data and results to their process and outcome data. 
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Section 1 (data) discusses what data you need to make decisions and where to find the data. Section 2 
(analysis) discusses how to understand the data once you have it. Section 3 (presentation) explains how 
best to present your analysis: how to tell your story. Section 4 (communication) describes how to 
disseminate the results and consider with others what the results mean. 

Concluding comments summarize how using data for decision making can improve services, service 
delivery, and, ultimately, outcomes for children, youth, and their families. Appendices provide resources 
for more information, including a list of sources for data and examples of data indicators (Appendix A) 
and an annotated bibliography for further reading (Appendix B). 
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Lj 


1. Using Data 

It doesn't matter whether you are a new agency head frustrated that you can't 
get reliable numbers of children in out-of-home care or a social worker 
perplexed by an apparent relationship between autism and physical abuse. In 
both cases you need to go through some kind of process to determine exactly 
what you want to learn and then identify the core data necessary to do so. Any 
decision can be informed by the data that explain the problem and will support the direction you want 
to take to address it (National Federation of Families for Children's Mental Health, 2009). 


There is a simple and universal answer to "what data do I need?" and that is, it depends on what 
question you are trying to answer. Understanding system intent determines the types of data collected. 
Systems participating in the system of care will need to make strategic decisions about what data will be 
collected based on a shared understanding of what they want to achieve (Kukla-Acevedo et al., 2008). 


Whatever you are trying to do—whether it is to create a new data system, improve an existing data 
system, design progress reports, or better understand the people being served in the system of care— 
you need to go through a five-step process: 


1. Articulate the questions you need to answer, being sure they are linked to the overall outcomes 
you hope to achieve. 

2. Identify the core data you need to answer the questions and collect only what you need and what 
will be of value, not just an added burden to collect. 

3. Determine whether the data you need are currently available. 

4. Define major gaps where data are not available, incorrect, incomplete, or don't make sense. 

5. Plan a strategy to fill the gaps and obtain the right data. 

The process of conducting all five steps can be formal or 
informal; based on a small core group or given as a charge 
to a much larger committee; supported by staff members 
in-house or contracted facilitators; conducted over a very 
short period, such as a morning retreat, or over a long 
process. 

Regardless of how you establish the process for defining the 
questions and identifying core data elements, there are a 
few considerations to keep in mind right from the start. 

First, leadership is critical—both in communicating an 
overall vision and philosophy of what is to be achieved and 
in ensuring the operational and practice rigor needed to 
develop and use reliable data to inform decision making. 

Data indicators that are established are based on the principles and values and serve as "building 
blocks" for holding systems accountable for achieving outcomes for youth (Casey Outcomes and 


Virginia: Commitment to Systems 
Transformation 

In Virginia, strong, committed leaders 
played an important role in 
communicating the values and principles 
that guided the practice model and system 
transformation to create a continuum of 
community-based services to support 
children and families. No meeting was 
held that discussed data without sharing 
the practice changes that needed to be 
achieved to improve outcomes for 
permanency and lifelong connections. 
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Decision Making Project, 1998). Leaders will need to continuously demonstrate and model at meetings 
at all levels of the organization that managing by data is the norm. The leader's job is to create a "safe 
place" for people to question what the data tells us and be open to exploring the meaning of the data. 

Second, key stakeholders need to be won over to the idea that numbers and other data are not useful 
solely for data analysts or other "data wonks" (Webster, Needell, & Wildfire, 2002). Don't default to 
data analysts or information technology (IT) staff when you identify the questions you are trying to 
answer and what you need to know. IT staff can help you to assess what data exist and where the gaps 
might be, but only those who are providing and receiving services, including children, youth, and 
families, can define the right questions. 

A change in perspective and an expectation that they will be at the table is crucial for all stakeholders to 
become data-driven and outcome-focused (Jamieson & Bodonyi, 1999.) Leaders, direct service workers, 
supervisors, managers, youth, and families should all be included in the process. The questions need to 
come from people engaged in the actual practice and service delivery. By managing by data, 
stakeholders can play an important role in asking not only what the current status is but "what practice 
changes are necessary to move the numbers?" Recognizing that significant change takes time; 
measuring the small incremental improvements over time ensures stakeholders stay engaged. 

Third, focus diligently on the right questions and the data needed to answer them. The right questions 
are going to be those questions that will help you determine whether you are achieving your priority 
outcomes. When you consider the outcomes you wish to measure, you may want to determine what 
works and what doesn't work. In that case, the question is not "did he or she participate?" but "how did 
participation in the program improve a child's success?" 

For example, let's assume you have determined that keeping children and youth safely in their own 
homes in the community, avoiding out-of-home placement in a residential facility, group home, or 
psychiatric hospitalization, is the primary goal. You will want to ask, what is our entrance rate of children 
placed in out-of-home settings compared with previous years? What are their presenting needs? How is 
the decision regarding placement determined? In a year or so, you are going to want to know how that 
picture has changed. In 5 years you are going to want to know that out-of-home placement is no longer 
the norm and that you have created a system of care adequate to support children and youth in their 
own home by providing access to community-based 
services, supports, and resources. 

A very insightful caseworker shared, "Data doesn't do the 
work; it just tells you what work should be done." However, 
don't get distracted by how you are going to transform 
practice to get a different result than you expect to find 
when the data are analyzed. That can come later. 

Don't agonize over trying to make the key questions and/or 
source of data perfect. They never will be. Inevitably, both 
the questions and the core data will evolve as broader and 
broader exposure to families, additional practitioners, 

advocates, and others occur, and as you learn from early efforts. Your initial questions will be refined, 
benchmarks will change, and data indicators will be modified. This iterative process will improve the 
quality of the data and inform decision making. 


New Jersey: Reflecting on the Data 

In New Jersey, the Commissioner of the 
Department of Children and Families has 
hosted annual leadership summits to 
provide an opportunity for managers to 
review progress made in key outcome 
areas. Longitudinal data illustrated 
through charts and graphs is used to 
discuss the priorities and areas of focus for 
the uDcomine vear. 
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It is unrealistic to expect large planning groups to reach 
total consensus on what is important to know and why. 

Beware of discarding critical questions just because they 
are controversial. For example, you might have a long¬ 
standing debate in your agency about the right balance 
between safety risk and family reunification. You may 
argue for a long period about how to raise questions so 
that you can test appropriate approaches. In the end 
there may be no resolution, and it will be tempting to put 
those questions aside. Regardless of the controversy, 
leadership needs to say, "just get the data." 

Different parts of your agency—and indeed different 
parts of any system of care—may have different key 
questions and core data needs. Different organizations 
within the system of care will have different goals and 
priorities and different data needs to demonstrate 
success toward those goals. For example, juvenile justice 
focuses on public safety, education focuses on academic 
achievement, child welfare focuses on child and youth 
safety and permanency, and mental health focuses on 
child and family well-being and functioning. In addition, 
what your financial manager and your clinical director 
need to know are not necessarily the same. Although, 
ultimately, the diverse data will need to be incorporated 
into a single, overall integrated data system, which is not 
the essential challenge. Instead, it is to identify the 
questions regarding the most important challenges facing your system, services, or child and family 
outcomes. Creating an organizational culture that fosters the discipline to be accountable for answering 
those questions is the greatest hurdle to overcome. 

Common Data Types and Their Uses 

When you are thinking about what type of data you need to answer any particular question, you should 
keep in mind that generally there are several types of data you will most frequently encounter: point-in- 
time, longitudinal, process measures, and outcome data. These different types of data have different 
uses and different users. 

Point-in-time data refers to exactly what it says: a specific point in time and what variables looked like at 
that point—a snapshot of one moment in time. Data is simple, timely, and easy to understand and use. 
For example, say that on May 31, 2011, the Department of Children and Families had 7,100 children in a 
placement out of the home. If a major goal is the reduction of out-of-home care, you could look at two 
separate points in time (today and 1 year from now) to determine whether there were any changes. 
However, point-in-time data to explain events that happen over time can be misleading and will provide 
a limited picture about the youth and families being served, how long they have been receiving services, 
and whether they are achieving their goals. 


New Jersey: Learning to Use Data 
for Decision Making 

In September of 2010, New Jersey's 
Department of Children and Families (DCF) 
announced a new fellows program 
designed to provide an opportunity for 
100 experienced staff to attend an 18- 
month seminar series in which they will be 
exposed to best practices in using data to 
support improved case practice and 
outcomes for children and families. DCF 
implemented an ambitious curriculum to 
help provide the participants with the 
tools and skills they needed to become 
effective consumers and users of data — 
and, most importantly, compelling 
storytellers about the impact new practice 
and practice reform was having on 
outcomes for children and families. DCF 
was very smart to recognize that being 
analysts and data and policy wonks was 
not what drew most people to social work 
as their profession. This curriculum was 
designed to give the "new social worker" 
the tools and skills necessary to enhance 
their practice in this information age. 
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Longitudinal data are collected over time. There are 
many methods for collecting longitudinal data, including 
time-series, cohort analysis, and statistical correlations. 
Longitudinal data, collected at determined dates or key 
decision points, will determine whether a pattern is 
discernible. Longitudinal data is often used to measure 
systemic improvements over time. For example, how, 
when, and why placements shifted from congregate to 
community-based care. You could also divide the 
population into cohorts (by entry into care in successive 
years) to see whether there was change comparing one 
cohort to the other. For example, understanding issues of 
disproportionality requires tracking the percentage of 
minority at key decision points throughout the system, 
including referral, placement, and discharge. 

Process measures measure performance of a process. 
They provide information that can be acted on quickly 
when a process provides data indicating something is or 
is not working as intended. Process measures are familiar 
to staff, relevant at the caseworker level, and current. For 
example, in June, 92% of children in child welfare custody 
had face-to-face contact with their caseworker. 

Outcome data are the "big picture" measures of system 
performance, especially when looked at over time. For 
example, the median length of stay for children who 
entered a system in 2003 was 12.4 months, and in 2009 
that median stay had declined to 9.5 months. 

Both point-in-time and longitudinal data can involve a 
variety of things: 

• Volume of activity: how many cases, how many 
children, how many referrals. 

• Descriptive characteristics: age, gender, ethnicity, 
employment status, economic status, family 
composition, adults and siblings in home, address, 
ID numbers (e.g., SSI, Medicaid). 

• Clinical/diagnostic data: functional assessments, 
diagnostic results, health assessments, treatment 
plans, treatment progress, treatment outcomes. 


• Status/case progress: where a particular case is at 

any time and at any point in the process (e.g., intake, diagnostics, temporary placement, referral 
to services). 


Virginia: Changing the Practice 

To focus on what was being accomplished, 
Virginia articulated a new vision for the 
agency about using data, one fully 
integrated with practice. According to 
Matt Wade, Outcomes-Based Reporting 
and Analysis (OBRA) Manager, Virginia 
Department of Social Services, "The 
practice of collecting and sharing data and 
information is a nonnegotiable part of how 
we continually learn and improve. We will 
use data to inform management, improve 
practice, measure effectiveness, and guide 
policy decisions." To support this vision, 
the agency created an OBRA unit. Their job 
was to begin designing outcome measures 
and producing reports that could inform 
management, improve practice, measure 
effectiveness, and guide policy practice. 

When OBRA initially produced 
management reports, as is predictable 
when you start any management-by-data 
effort, field staff were highly skeptical. 

The centrally produced reports had 
numbers they didn't recognize and that 
couldn't be reconciled with their own 
local data. Some of the results seemed 
counterintuitive. Even though OBRA had 
a very small staff, they made a critical 
decision to go out to every field office 
and sit down with all staff and go through 
their numbers in some detail. It produced 
epiphanies for many field offices. Not 
only did they find that the highest 
percentage of errors was the result of 
coding problems in the local office (an 
almost universal problem in any system), 
but clarification of the terms and 
definitions used to design the report 
made it possible for local staff to 
reconcile their own data. The data even 
became useful! Virginia's ultimate goal is 
to have both the data report and the data 
sets available to every staffer via the Web 
for transparency and availability to all, 
including policymakers and the media. 


8 





Putting the Pieces Together: Guidebook for Fact-Based Decision Making to Improve Outcomes for Children and 
Families 


• Disposition: final placement, reunification, permanency outcome. 

• Financial: revenues and expenses associated with cases, interventions, and dispositions. 

• Research: evidence developed through various types of research about the efficacy of various 

interventions and the correlation between factors influencing a given case or group of cases. 

Although these categories are not entirely mutually exclusive, they are helpful to use as a checklist when 
trying to match data needed with questions to be answered. 

Below are some examples of key questions that a system of care might ask and the kinds of data that 
would be helpful to answer the questions: 

• Can long-term psychiatric hospitalization be significantly reduced? 

- Volume: How many hospitalizations (entries) were there in a given period? 

- Descriptors: What was the age, sex, ethnicity, family status, living circumstances, and 
education of those hospitalized? 

- Clinical: What was the functionality of the child/family at referral? Were there other 
medical/mental health diagnoses or presenting problems? What was the functionality or 
diagnosis at each subsequent stage? 

- Process/status: What was the origin of the case/referral source? What was the status in other 
human service systems? What was the treatment plan? What was the median length of stay 
for those who entered care in a given year? 

- Disposition: What was the behavioral and social functioning at completion of an intervention? 
Was there permanent connections made with family, guardian, adoptive parents, or others? 

- Financial: What was the total case cost from referral to disposition? What were the revenue 
sources for costs? What are the costs at each stage of processing/intervention? 

- Research: What were the alternatives to psychiatric hospitalizations for children? What were 
the existing community-based services for youth with serious emotional challenges? Were the 
psychiatric placements disproportionately occurring for various ethnic groups? 

Another question might be: 

• What percentage of juvenile first-time offenders can be diverted from the court system? 

- Volume: What is the number of first-time offenders? What numberentered into diversion 
services? 

- Descriptors: What are their ages, genders, races, family status, living circumstances, and 
education levels? 
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- Clinical: What were the circumstances of the arrests/charges? What were the findings at the 
initial screening? What are the nature and stability of their family/community relationships? 

It is worthwhile to explore resources that identify key indicators that the system can track. Appendix A 
(Sources for Data and Examples of Data Indicators) identifies a few of those resources. 

Mapping Your Questions and Data Needs 

It is essential to know the origin or basis of your data when you are trying to provide information or 
make a point. 


Recently, a group of key stakeholders met to discuss services with representatives from the judiciary, child 
welfare, health and mental health, and external advocates. A member of the stakeholder group put up a slide that 
said. "Our current caseload is 1,000." The member then proceeded with a number of other slides about programs 
and services. 

While some general discussion ensued, no one raised any questions about the 1,000. What was it? 1,000 kids? 
1,000 cases? A duplicated or unduplicated count? At a specific point in time? On average, over a period? 
Cumulatively for the year? Obviously, it meant nothing without these qualifiers and definitions. Good instinct to 
use a number, but it needs to be explained to be meaningful. 


Remember that the information you need or want may be either quantitative (things you count or 
statistically correlate) or qualitative (things you observe or believe). Any type of human services data 
could be either. Both are often necessary. In recent years, human service systems have found that 
formal "quality service reviews" are a key part of their quality improvement systems and yield 
considerable qualitative data. When combined with quantitative data, the ability to focus in on areas of 
strengths and weaknesses in performance contribute significantly to achieving better client outcomes. 


New York City's Administration for Children's Services is a perfect case in point. Once a week senior staff and 
relevant caseworkers meet—with the director present—to review critical performance measures and review one 
or more current cases. The indicators provide the quantitative data, and the case presentation provides an 
opportunity to explore qualitative and quantitative issues. Both types of information are essential to 
transformation of practice and corresponding policies and procedures. 

Source: Jan Flory, New York City Administration for Children's Services Deputy Commissioner, Division of Child Protection. ChildStat: Leading 
systems-level improvements based on case-level experiences, What Works for the Workforce: Leadership Competencies in Action-A National 
Webinar Series on Leading Change to Strengthen the Child Welfare Workforce. Sponsored by National Child Welfare Workforce Institute, 
September 21, 2011. 


Using a Matrix to Organize Your Questions and Data Needs 

When you are conducting a process for defining the questions and identifying core data, a matrix can 
help organize your thinking. Anytime you need to sort out "what do we need to answer?" and "what do 
we need to know to answer it?"—developing a matrix can get you off to a good start. 

A matrix could list data types across the top and the questions to be addressed down the left axis. Not 
all types of data are required for every question, so every cell need not be filled (Table 1). 
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Table 1. Example of a Data-Needs Matrix 



Current Month 

Year to Date 

Critical Data Element 

Number 

Percentage 

Number 

Percentage 

Assignment: How many referrals are 
assigned promptly? 

Referrals assigned within 10 days. 





Length of Stay: How long do youth of 
different ages stay in care? 

Current enrollees (ages 0-5, 6-12,13 
and older) length of stay less than 

12 months. 





Change in Child and Adolescent 

Functional Assessment Scale (CAFAS): 

Do youth demonstrate improvement 
while in care? 

Percentage of those enrolled with 
change >10 points at 6 months. 






Adapted from "Critical Data Dashboard," by Child and Adolescent Treatment Services, Erie County Department of Mental Health, New York, 
March 2010. 


Alternatively, the left axis could list key decision points in your process (e.g., referral, intake, screening, 
assessment, investigation, custody, placement) and the data cells would include what you need to 
better understand the children and families served in that phase in the process. 

The total number of children being tracked in these charts may be small and so caution should be used 
in interpreting the results when comparing cases from one month to another. The results may indicate a 
wide variation in the percentage of change. For instance, 1 of 3 referrals assigned in 10 days is 33%, but 
2 of 3 referrals assigned in 10 days would be 66%. 

Where Can You Find the Data You Need? 

After you have settled on key questions and core data needs, you can search data systems and records 
available to you and determine how many of the core data elements exist and are accessible in some 
form. Don't limit yourself to just the case management or business data system that you use. Think 
about the data you have identified and strategize about where and how that data could be acquired or 
collected. In a fact-based, decision-making culture, operational, financial, policy, and regulatory data are 
all treated as assets of the agency and system, not the purview of a single office. 

To get the requisite data, you may need to: 

• Extract data from multiple data systems used in your own agency or program 

• Generate a data set from manual records or case notes 

• Add additional data elements to your case management screens to begin collecting particular 
data elements 
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• Explore the feasibility of obtaining data from other 
agencies or programs. At the state, county, and local 
levels, there are considerable data resources that are 
used for various purposes. For example: 

- Child care coordinating council (available slots by 
age groups and the cost of slots) 

- Head Start (community needs assessment) 

- Social services (children in foster care) 

- Health department (immunization, WIC 
participation, early intervention) 

- United Way (community assessment report) 


• Access national databases, including the U.S. Census, 

as well as private and nonprofit research institutes and centers to get data sets for comparison 
(Appendix A). 

It is never impossible to find specific information, even if the only solution is to begin collecting it now 
and accept that historic data may not be possible. "We don't have it," "we can't do it," or "it's not 
possible" are simply unacceptable answers. Be creative, be smart, and think about multiple sources for 
the same data elements. Even consider entering manual data into a database if that is the only way it 
can be done. For example, Medicaid numbers may be available only in hard copy from the state agency. 

Coping With Data Limitations 

Just because you have found the data you think you need doesn't mean it is guaranteed to be accurate. 
Like virtually every other aspect of human services, common sense and good judgment are as important 
to using and interpreting numbers as they are to good counseling and compassionate care. 

A clear and consistent timeline for reviewing periodic reports, submitting corrections, and seeing revised 
results is helpful. The process should be explicit about who is responsible for entering data, assuring 
data accuracy, and correcting errors when found. Opportunities need to be provided to bring data- 
reporting issues to light and give workers a chance to enter missing information and correct 
documentation errors (Jacobsen, 2007). A strong Help Desk or other supports for the staff who enter 
data will go a long way in helping users and overcoming data limitations. 

Data Systems 

While having some data is better than having none, weak or limited data systems will result in pressure 
for better and better solutions. Don't disregard national or proprietary data systems that may be 
available and may save significant time and effort. Some states, for example, have taken advantage of 


New Jersey: US Census is Important Data 
Source 

As New Jersey trained field staff on the 
use of data through the "Fellows" 
program, demographic information, often 
obtained from the U.S. Census, was a 
powerful place to set a context for the 
population being served. Comparing 
incomes, percentage of income used for 
housing, and the age/race/ethnicity of the 
population in a county with that of the 
state or neighboring counties, often 
highlighted specific needs or issues. The 
Census data was easily understood, 
particularly by those who were venturing 
into the data world for the first time. 


Use Tools That Make Data Timely, Accessible, and Interactive 

Casey Family Programs invested in a product called SafeMeasures to be used across Virginia so state data could 
be uploaded twice a week and shared with counties who could see it, use it, and correct it if it is not accurate. 
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SafeMeasures®, a measurement system developed by 
the Children's Research Center, National Council on 
Crime and Delinquency (http://www.nccd- 
crc.org/nccd/initiatives/safemeasures.html). Several 
states have used the Results-Oriented Management 
System developed by the Kansas University School of 
Social Welfare (https://rom.socwel.ku.edu/ 
ROMTraining/). Another system used by some states is 
the Integrated Database of Child and Family Programs 
developed by Chapin Hall Center for Children. This 
database incorporates and links data from multiple 
human service agencies. 

Accessibility of Data 

Fact-based decision making cannot be transformative for 
the organization or across a system of care unless people 
understand and believe in the facts. Breaking data down 
for availability at the state, regional, office, and/or unit 
and worker levels encourages use of reports, provides 
transparency, heightens interest, and even promotes a 
sense of healthy competition. Such data clarifies where 
performance is strong, where improvement is needed, 
and where action should be taken to strengthen practice 
and build on strategies that work (National Child Welfare 
Resource for Organizational Improvement, 2010). 

To help people both understand and believe the data, 
you have to: 

• Ensure accessibility of the data, not just at the 
summary level but down to the individual data 
reflecting each direct service worker's own 
caseload or practice. 

• Help staff understand how data is entered and 
compiled and how they can validate the data if 
something looks amiss. 


• Create a credible process for correcting coding or other administrative errors and foster a 
willingness and capacity to correct structural or organizational deficiencies that undermine data 
accuracy. 

• Hold safe (nonpunitive) forums to promote collective responsibility for data integrity and to have 
the opportunity to discuss implications for individual, group, and overall agency or system 
practice. 


Getting the Right Data 

A juvenile justice facility had a long¬ 
standing process for recording and 
reconciling census data. They maintained a 
daily census, but (a) it was always taken in 
the morning and therefore did not include 
any children released or transferred that 
day, and (b) it recorded a discharge or 
transfer only when the case file for that 
child was closed in the case management 
system by the individual caseworker. It 
was not uncommon for there to be a lag of 
1 or more days between the actual 
discharge/transfer and the closing of the 
case file. Further, these actions were 
generally not fully reconciled (for the 
previous month) until the third week of 
the current month and not necessarily 
reflected in the data system until the 
month-end census. Consequently, when 
each prior month's "results" were 
published on the 10th of the current 
month, they did not reflect the actual 
discharges and transfers still 
administratively pending. 

Because of this long-standing process, the 
monthly reports often reflected census 
statistics in excess (sometimes 
significantly) of the actual facility capacity. 
Managers and caseworkers were 
frustrated because the numbers simply did 
not reflect reality, and outside observers 
and advocates were concerned about 
perceived overcrowding problems. A 
change in policy and a change in the 
database—recording the discharge or 
transfer when the child physically left the 
facility—was what was needed. 
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Summary 

To know what data you need, you have to determine what questions you are trying to answer and the 
general and specific information needed to answer them. Once you know the questions, finding the data 
you need requires organizational and policy flexibility, familiarity with information systems used in the 
agency, a willingness to digitize manual records and information if necessary, and, if appropriate, a 
willingness to explore data sources outside of the organization. 

The use of data is a function of its quality. The more transparent the data are throughout an 
organization, the greater the potential for data accuracy. Across a system of care, there will be a 
collective responsibility for data integrity—it becomes everyone's job. 

These are the most important questions you need to ask yourself and your colleagues as you begin to 
create a fact-based, decision-making culture: 

• What are we trying to accomplish—our goals and stated outcomes? 

• What will indicate whether we have accomplished our goals? 

• How much of the data do we have, and is it accessible and usable? 

• For data we don't have, how can we begin collecting it? 

• What internal administrative training, IT system, policies, or communication barriers do we need 
to overcome to make the process of identifying and collecting data better? 
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2. Conducting Analysis 

You don't have to be a trained researcher, analyst, or statistician to be able to 
use the data you are collecting to analyze a problem. Data analysis is simply 
compiling, evaluating, and presenting data to support and illustrate answers to 
questions that have been raised. The goal is to highlight useful data and suggest 
conclusions to support decision making. 


In general, the simpler the analytic method the easier it is to present and communicate the results. 
While the tendency might be to go "wide" with data to look at many things, realistically going "deep" to 
examine issues in depth is more productive. Qualitative tools—such as case reviews, focus groups, and 
interviews—enable you to further analyze a measure or an issue. 


Once you know the questions you are trying to answer and the data you can generate, you can define 
your measures and actually undertake the calculations, such as: 


• Totals/sums: total cases, total children served. 

• Ratios: rate of placement per 1,000 children in the population, number of girls referred to total 
referrals, number of juvenile offenders admitted with prior offenses to total admissions. 

• Averages: median (the number in the middle of a list of numbers arranged from lowest to 
highest), mode (the number in a list of numbers that occurs the most often), and mean (the sum 
of all data divided by the number of data—the average length of time in care, average census per 
day, average results of functionality assessments at intake, etc., which can be affected by an 
outlier or number far outside the normal range and therefore should be used with caution). 

• Variance analysis: nominal or average changes in a given factor from period to period or from a 
baseline such as the approved budget to the actual result (e.g., year-to-date spending). 


As you probe more deeply into the relationship between the services offered and the short, 
intermediate, and long-term outcomes for children and families, you will also encounter a variety of 
statistical tests designed to measure the causal relationship between different variables and/or the 
relative statistical significance of different results. 


Types of Analysis 

There are many ways to analyze data. We describe some of the most frequently used. 

There are two kinds of longitudinal analysis used most frequently: time-series and cohort analysis. Time 
series (Figures 4 and 5) analyze different variables at predictable points in time over a period 
(e.g., compare expenditures over time for out-of-home placements in one county versus the entire state 
or track reductions in psychiatric hospital stays over time). Time series are used to identify seasonal 
relationships, cyclical patterns of behavior, or other trends or patterns that might affect policy or 
procedure with respect to practice. 

Cohort analysis has also become popular in child- and family-serving agencies. Cohort analysis looks at 
specific groups of people who share some major factor in common (e.g., all children entering child 
juvenile facilities in 2011). You might look at all children who received mental health services between 


15 




Putting the Pieces Together: Guidebook for Fact-Based Decision Making to Improve Outcomes for Children and 
Families 


ages 5 and 8 who were 
subsequently admitted to a 
psychiatric hospital between 
ages 15 and 20. Cohort 
analysis can be used to 
determine whether 
interventions and service 
delivery resulted in different 
outcomes for different 
cohorts, and why. 

A incidence, prevalence, or 
gap analysis can be 
important tools for 
identifying need and 
available services. For 
example, a prevalence 
analysis in New York City 
revealed that there are 
significant numbers of 
children within community 
districts in New York City 
with mental health needs, 
which can be used to identify 
necessary treatment slots. 
Table 2 illustrates a portion 
of the data collected from 
that study. 


Over time, the population in 
a system of care, costs, and 
services will change. Based 
on the rates of change, you 
can predict future rates. 

Predicting growth helps 
systems anticipate the need 
for increasing or decreasing 
human services. To find the 
annual percentage change, you need to know the initial population, the ending population, and the 
number of years. To calculate percentage increase, first subtract the greater number from the smaller 
number to find absolute change. Divide the absolute change by the original number and multiply your 
answer by 100. 

Forecasting models attempt to predict both the volume and the outcome (e.g., the number of children 
likely to be admitted to psychiatric hospitals in the next year). They are also frequently used to predict 
cost and revenue in child-serving agencies and systems. A forecasting model uses actual data from prior 
periods to predict future behavior. Some models assume that future behavior will continue to mirror 
current behavior. They may be based simply on what happened to one factor each month of the prior 


Figure 4. Time Series Analysis of Expenditures 

Total Out-of-Home Expenditures 



1982 1983 1984 1985 1986 1967 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 

Fiscal Years 

•Based on the CPI, Populations under 18 [ H>-Santa CfUZ California | 


Figure l.Total Out of Home Expenditures Through June 30, 2001. Adapted from 
"Comprehensive interagency System of Care for Children and Youth: Measuring Outcomes of 
Collaboration: Seventeen-Year Report, July 1,1989-June 30, 2006," by Santa Cruz County 
Mental Health, p. 1. 


Figure 5. Time Series Analysis of Hospitalizations 


State Hospital Utilization Days Through 6/30/06 



prior to 

1989 Fiscal Year 

Figure 8. State Hospital Utilization Days Through 6/30/06. Adapted from "Comprehensive 
Interagency System of Care for Children and Youth: Measuring Outcomes of Collaboration: 
Seventeen-Year Report, July 1, 1989-June 30, 2006," by Santa Cruz County Mental Health, p. 8. 
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and current year (e.g., total expenses). They then project that behavior going forward and may or may 
not take into account other factors such as change in prices (inflation) or known changes in policy (such 
as a reduction in the rate of federal reimbursement for administrative expenses). They can also be used 
to examine policy effect on systems (such as what would happen if I reduce my intake by 5%). 


Table 2. Prevalence of Mental Health Disorders Among New York City Children Ages 5-17 


Total Number of 5 to 17 


Serious Emotional 
Disturbance: Total 


Mental Health Disorders: 


Community Districts 

Year Olds 

Prevalence (10%) 

Total Prevalence (20.0%) 

Staten Island 

Willowbrook 

32,568 

3,257 

6,514 

South Beach 

23,294 

2,329 

4,659 

Tottenville 

29,908 

2,991 

5,982 

Total 

85,770 

8,577 

17,154 


New York City Total 

1,343,715 

134,372 

268,743 


Adapted from New York City's Children and Mental Health: Prevalence and Gap Analysis of Treatment Slot Capacit 
Report, January 2012. 


More complex and technical statistical analyses are necessary to explore the relationship between 
intervention and outcomes. There are many studies that demonstrate the evidence that different 
programs and interventions are successful in addressing particular mental health issues and behaviors 
(e.g., evaluations of multisystemic therapy [MST], parent-child interaction therapy [PCIT], or trauma- 
focused cognitive behavior therapy [TF-CBT]). However, it is important to recognize that an evidence- 
based program will only yield reliable outcomes if it is faithfully executed with the intended population 


While understanding the meaning of analysis is critical, it is also important to balance the research 
results against experience and common sense. If the results just don't make sense, spend time with 
colleagues and/or the researchers and talk it through to break down the meaning behind the numbers. 
Consider the implications of the findings for you, your practice, your agency, or the system of care. 


Incorporating Standard Analyses Into Your Daily Practice 

Agencies and systems that are trying to establish a fact-based, decision-making culture often start with 
periodic monitoring reports. These include a standard and consistent set of measures or specific 
analyses. There are numerous ways to select the measures or analyses that you want to routinely 
produce. Different stakeholders will want to know different things. There is a popular saying that "what 
you measure is what you manage," and the experience of many indicates that it is true. "If you don't 
identify it, it won't get counted. If you don't count it, it won't get measured. If you don't measure it, it 
won't be in the budget. If it isn't in the budget, nothing will happen." Ken Martinez, TA Partnership. 

If you are just starting to use data in your practice, the most important thing is not that all the measures 
are exactly right, but that the reporting gets started. Useless measures will rapidly fall by the wayside 
because they won't help anyone inside do a better job and they won't help anyone understand how well 
they are performing. Time and use will let you know if you need to modify or enhance your measures. 
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There are important caveats to remember when you are 
starting to develop a monitoring system, dashboard, or 
similar data tracking tool: 

Measures should be closely aligned with the priority 
outcomes you hope to achieve. If you are trying to 
significantly increase the use of community-based 
resources in your practice, then you have to keep your 
eye on trends in community referrals and whether those 
services are actually being delivered. Tracking the 
number of licensed children's mental health providers 
under contract to your agency, for example, tells you 
nothing about whether you are making referrals and 
whether those services are actually being delivered. 

What you track needs to match up with what you are 
trying to accomplish. 

Measures shouid be simpie and transparent. It won't 
help anyone if the measure is so complicated and opaque 
that people not only can't figure out what it is supposed 
to measure but haven't got a clue how it was calculated. 

A really strong monitoring system will rely on clear 
measures and ensure that users understand the data 
values used, dates the data were extracted and/or 
analyzed, the organizational entity that prepared the 
data, and the methodology for collecting the information. 

Assumptions or definitions that explain the measure 
must be easily understood. Experience show that when it 
is possible to share the raw data it helps builds 
confidence in the analysis. 

Data iimitations or weaknesses shouid aiso be 
transparent so that users can baiance them against the 
resuits. Data will never be perfect. Sometimes it is 
difficult to change past practice and behavior so that 
common errors are avoided. Knowing about potential 
weaknesses in the data brings attention to them and may accelerate people's willingness to correct the 
policy or practice. 

Measures shouid be actionabie by particuiar users. What is being measured should be able to help a 
worker, supervisor, manager, or client to do something differently and should be informative and 
relevant to their work. Share measures that are most relevant—and actionable to certain departments, 
systems, and levels of staff. 

Measures are dynamic, not static. It is fine to measure change or refine measures over time provided 
that the changes are clearly explained in subsequent reports. Analyzing your practice is a learning 
exercise, not simply a reporting one. As you align goals with results, you will discover that you may have 
missed some key factors when you started, or you may have defined certain measures in a way that is 


Defining the Most Important Measures 

Virginia identified the state's top priorities 
and focused on demonstrating progress on 
them. To focus on performance, the 
transformation effort centered on six key 
indicators—data points that easily could be 
tracked and were clear signs the system 
was turning in the right direction. Those 
indicators included: 

• Total number of children in foster 
care 

• Number of children in family-based 
care, including in the care of a relative 

• Number of children in congregate 
care 

• Length of stay in foster care 

• Discharges to permanence 
(connecting children to family rather 
than allowing them to age out of the 
system without connections) 

• Number of children who returned to 
the system because of repeated 
maltreatment 

Virginia set directional goals for each of the 
measures—for example, they wanted 
congregate care placements to go down 
and placements in family-based and kinship 
care settings to go up. These results were 
produced monthly and were made 
available at the statewide, countywide, and 
individual-office levels. 
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not as accurate as it might seem. Each individual agency 
needs a degree of flexibility in its measurement system 
to accommodate this learning. 

Developing Data Dashboards to Track Progress 

A dashboard is simply a shorthand version of key 
measures that is easy to read and understand, widely 
disseminated, and recognizable to people at all levels of 
the organization. As a colleague noted recently, 

"Anybody can design a dashboard. You can copy 
someone else's, gin up a set of measures that make 
your policymakers happy, or just reconfigure standard 
reports that you have been preparing for years." But 
this is not the point. The key is to determine what you 
are trying to accomplish as an agency or system, and 
then align measures with those goals so that 
everyone—including children and families—can be 
engaged in defining what is most important, tracking it 
over time through their own experience, and 
understanding the impact on the system as a whole. 

A dashboard can: 

• Be a formal or informal process 

• Evolve over time 

• Involve a small group of designers or a very large 
group of participants from all disciplines in the agency or system 

Dashboards are most effective when they reflect a small set of critical facts—ideally just three or four 
measures that are at the heart of the mission of an organization and demonstrate performance of the 
agency in achieving its mission. These facts will be regularly produced, widely disseminated, and easily 
understood by multiple stakeholders. 

Benchmarks and Comparative Analyses 

You may also frequently encounter pressure internally or externally for the performance of your agency 
or system of care to be compared with other jurisdictions. 

Benchmarks, best practices, and other national or professional standards can be informative as long as 
the limitations are clearly understood. Many states, have found it more relevant to compare their 
performance for current years against past years, especially using longitudinal data. 

Comparative analysis among different human services systems is difficult. You may not be able to define 
"apples to apples" easily. Different terminology, significantly different practice models, or disparate 
cultural norms can render the analysis incorrect. Also, things that may be best practice in one 
environment may not even be feasible in another. 


Know Your Facts 

Former Louisiana Child Welfare Director 
Marketa Garner Walters describes the 
importance of having key data accessible: 

As a child welfare director dealing with the 
aftermath of Flurricane Katrina, I learned 
that you need three to five numbers at your 
fingertips at all times for policymakers, 
administrators, the media, and the public. 
You need to speak with authority about 
those numbers, and they need to be: 

• Accurate and a correct accounting of 
the data 

• Up to date 

• Provided to you on a regular basis. 
(Staff need to be held accountable for 
doing this.) 

What those numbers are depends on your 
situation, your needs, and your gaps. For 
Louisiana the data we needed daily was the 
total number of kids in welfare custody, the 
number of those in home services, and the 
number in congregate care, by race, gender, 
age, and region of the state. 
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A Hierarchy of Measures 

A county has come a long way in establishing a system of care. One of the system's primary goals is to keep a 
larger percentage of seriously emotionally challenged children in the community with their families. They have, 
again over a fairly long period, settled on the definition of "seriously emotionally challenged," and all points of 
intake in the system now use a common assessment tool to make that determination. 

The county board wants to know, overall, whether they are succeeding in having a larger percentage of these 
children maintained in the community and with their family. They also want to know whether the cost of this 
policy is greater or less than the former practice of placing a very high percentage of these children in out-of- 
county residential care. 

Administrators in the systems want to know the extent to which in-county placements increase the probability of 
a permanent home placement or not. 

Clinical staff are concerned about whether the in-county placements are efficacious (i.e., are the children's 
behaviors improving as measured by testing functionality?) and the level of support that has been required to 
keep children in their own home when possible. 

Under the broad rubric of "seriously emotionally challenged" children a hierarchy of measures is provided that 
informs each of these stakeholder groups. A dashboard is created that starts with the broadest measures for the 
policymakers and drills down to detailed measures of clinical progress and individual staff decision making. 


Nevertheless, almost anyone can come up with examples of how other people do things that appear 
better than what your own system is reporting. Just as it is important to educate the consumers of your 
measures and data about the methodology and data sources you are using, it is important to educate 
them about the detailed differences—the "apples to oranges"—that can be present in benchmarking 
and comparative analysis. 

Summary 

You don't have to be a math wizard to be an effective consumer of data in your practice. You do need to 
demand that the analysis be transparent and the methodology be clear so you can be confident that the 
results are accurate. 

Measures used in dashboards, scorecards, monitoring reports, or other periodic assessments of progress 
are data analyses. Make sure they are clear, simple, and objective in their narrative conclusions. Be 
flexible and update or refine your analysis as you learn more about what is happening in the system of 
care and its relationship to child and family outcomes. 

Using data to manage can be transformative if the data are: 

• Clearly linked to overall goals 

• Simple and transparent 

• Actionable by the stakeholder groups to whom they are tailored 

• Consistent enough from report to report to become easily recognizable by all participants in the 
system of care 
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What do you need to ask yourself when you are choosing the analytics or measures that will be most 
important to your system of care? 

• What are the most important things we are trying to accomplish? 

• What is the simplest and most straightforward way to measure whether we are making progress 
toward those accomplishments? 

• How are we going to guarantee the most accurate analysis of the data possible? What will we do 
to ensure that the data and the meaning of the data are accurate? 

• Are we sure that we understand the methodology of results on which we are basing policy? 

• Is the analysis of our data actionable? Can it impact what people do in the organization? 


"If you don't identify it, it won't get counted. If you don't count it, it won't get measured. If you don't measured, 
it won't be in the budget. If it isn't in the budget, nothing will happen." Ken Martinez, TA Partnership. 
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3. Telling Your Story Through Presentation 

Presentation is about using the data to tell your story and about the children and 
families you serve. Consider the difference between (a) telling a legislator that 
families you know are unable to access services and (b) showing a legislator that 
80% of the 3,000 families you polled stated they were unable to access services 
(Schrepf & Brown, 2011). 

For the most part, it is no longer necessary to write special programs for standardized reports, as many 
can be created or generated in spreadsheet applications (like Excel) with a fairly limited amount of 
expertise. Many applications have presentation options that are "built-in." You can use a database to 
develop your analysis, and then use one of the predesigned table formats, graphs, or charts to present 
results clearly and professionally. 

Contemporary applications can also help you maintain data integrity in your reporting. If the analysis 
and presentation are fully integrated, it is much more difficult for you to miss a critical number or fail to 
catch an erroneous formula in a cell. 

Nothing will undermine a management-by-data effort faster than reports that are incomprehensible, 
baseline data that are contradictory from month to month, or formats that change every time they 
come out. How you present the data and how you maintain quality control are critical. 

In thinking about presentation, keep in mind the following key considerations: 

Clear and simple always trumps complicated and dense. Whether you are using graphics or numeric 
tables, make them as clear and simple as you can without losing the importance of your analysis. Design 
presentations for your average viewers, not statisticians and economists—and make sure that they will 
understand the data. The Erie County system of care actively engaged families in both the design and 
the implementation of the data collection process and the reporting system to present data that are 
most compelling in reallocating resources to better meet the needs of families. 

Settle on a standardized format as soon as possible. Changing how a key dashboard or monitoring 
report looks from month to month can be quite irritating. Although you might not get it right the very 
first time, try to settle on a recognizable standard as quickly as possible. You want anyone, anywhere in 
the system of care, to look at a screen or bulletin board and immediately recognize the signature design 
of the dashboard. Use a standard time frame on your reports and specify whether you are reporting on 
a monthly, quarterly, or annual basis. Remember there are calendar years, state fiscal years, and federal 
fiscal years, so indicate which you are using when reporting an annual number. 

Avoid “data-speak" in expressing the key results. Which would you rather read? 

Result A: The result of calculating a ratio with specific gender as the numerator and total new 
admissions in May as the denominator, resulted in a finding that 80% of new cases in May were girls. 

Result B: 80% of new admissions in May were girls. 

You can have a footnote, attach a methods sheet that shows the formula for each measure, or provide a 
guide to the dashboard on a central website. 
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To help ensure data quality, link your tables and narrative directly to the underlying data and 
calculations as much as possible. It is always better to have your data and analysis linked to tables and 
presentations directly. This is easily done in most spreadsheet applications. By linking, you can be sure 
that changes in the raw data or calculations will automatically roll through your presentation. It is easy 
to have an erroneous formula or number (e.g., using March admissions for the April report), and 
catching them before publication often doesn't happen. Use the power of the software to help you 
maintain this quality control as much as possible. 

Create opportunities for an interactive review of the results in your presentations where possible. 

People love to create "what if?" scenarios. There is nothing that people like to do more than test the 
data themselves or imagine how outcomes might have been different if their pet theory were in place. 
Simple what-if models are relatively easy to design. Pivot tables can provide the most powerful 
interactive tools as they automatically sort, count, total, or give the average of the data stored in one 
table or spreadsheet and display the results in a second table, showing the summarized data. The user 
can change the summary's structure by dragging and dropping fields graphically. A little investment in 
training so that users can take full advantage of the software you have available can yield epiphanies 
among practitioners and others. 

For example, you might develop a budget model that projected cost based on the number of cases 
entering the system, the initial placement type (e.g., in-home, out-of-home family setting, out-of-home 
congregate care), and the provisions of children's mental health support services in the various settings. 
The model can be set up so that users can alter any of the variables driving costs. What if the ratio of in- 
home to out-of-home care were flipped? What if the number of referrals for mental health support 
services doubled as a percentage of total caseload? What if the number of cases entering the system 
could be reduced by 15%? In each case, what happens to the budget for the upcoming year? 

These models can often help users and their projected audiences internalize the relationships between 
complicated factors in a way they cannot understand otherwise. 

Be careful to present the results objectively. You want to stimulate discussion, but you want it to come 
from the numbers, not from something subjective in what you say about them. The importance of 
consistent reporting is that it should stimulate reflection and discussion about the relationship between 
what people are actually doing and what is being accomplished. People throughout the system should 
be able to form and express their own opinions about what the results mean on the basis of the data 
being presented. 

Never change a baseline or critical reference number from one report to another without being explicit 
about the reason for the change. For example, say the May dashboard reports a baseline from 1 year 
ago of 752 admissions in April. When the June report comes out, the April admissions are reported as 
735, changing the marginal, month-to-month trend in admissions for that quarter. What happened? A 
footnote explains that errors in coding in April of the prior year double-counted 17 cases, thus 
overrepresenting the number of cases that were admitted that month. A supervisor perplexed by the 
much larger admission volume for her team discovered the error. The footnote went on to explain that 
the error was corrected and the quality control that had been in place was improved so that it wouldn't 
be repeated. Changing baseline assumptions and numbers can undermine the credibility of the analysis 
and the willingness of overseers to accept the results and conclusions. 


23 



Putting the Pieces Together: Guidebook for Fact-Based Decision Making to Improve Outcomes for Children and 
Families 


Encourage people to look for and report errors In logic, math, or Interpretation. Errors in analyses are 
inevitable. The more you can encourage users and stakeholders to highlight problems if they find them, 
the better the reports will be over time. The goal is to consistently produce high-quality tracking results. 

Display the results In multiple forms. 

People absorb information in 
different forms. Some like numbers, 
some like pictures, some only retain 
facts when they hear them orally, 
and still others need to write them 
down to remember them. 

Presentation is about finding forms 
of transmitting knowledge that can 
be internalized by the broadest 
possible group. Experiment with 
different graphs, table formats, and 
even an audiovisual PowerPoint 
presentation or video that walks 
through the results and can be 
accessed when convenient by staff. 

Figure 6 displays samples of different 
types of graphs and charts to 
illustrate trends in teen pregnancy 
across the state. Some focus the eye 
(and analysis) more on the trend 
within the county, while others more 
readily support cross-county 
comparisons. 

The bottom line is that the data 
display will be critical for encouraging 
the usability of the data findings. 

Combining the visual presentation, 
along with a data champion or key 
contact person who can clearly 
present the data, will ensure the 
findings are used in a way that will 
make a difference. 

Summary 

How an analytic report reads is often 
as important as the results 
themselves. You need a culture and 
environment where everyone at 
every level accepts mutual 
responsibility for data quality and 

interpretation. Welcome people who report errors or mistakes. Correct them quickly and gratefully. 


Figure 6. Samples of Different Types of Graphs 

Adolescent Births by Age, 15-17 years 

Rate/1,000 females ages 15-17 years 
Motion plot data 



Adolescent Births by Age, 15-17 years 

Rate/1,000 females ages 15-17 years 
Bar graph data. Year by Region 
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Adolescent Births by Age, 15-17 years 

Rate/1,000 females ages 15-17 years 
Plot Data forYear(s) and Region(s) Selected 

Region 


1998 

1999 

2000 

2001 

2002 

2003 

2004 

2005 

2006 

2007 

2008 

19.4 

17.4 

17,6 

16.3 

16.0 

15.7 

14.9 

15.1 

12,7 

14.0 

11.7 

14.0 

21.9 

13.5 

14.8 

15.1 

14.7 

13.2 

14.9 

10.6 

11.1 

14.7 

8.3 

9.4 

8.0 

6.4 

6.2 

6.7 

7.6 

* 4.4 

5.2 

4.6 

6.8 

24.7 

20.3 

20.8 

17.8 

17.0 

19.3 

18.1 

14.4 

14.3 

14.6 

13.2 


Graph illustrates adolescent pregnancy rates in New York State and recommends 
the goal of reducing pregnancy to no more than 28 pregnancies per 100 females 
aged 15- to 17-years old by 2013. 

Adapted from "Graph Builder: NYS Kids' Well-Being Indicators Clearinghouse 
(KWIC)," by Council on Children and Families and Cogent Technologies, Inc., 2003- 
2012 . 
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Objective reports should communicate effectively to people with very different learning modes—visual, 
numerical, etc. Seek feedback and experiment until you have a consistent and recognizable format that 
the majority of participants endorse. 

What are the main questions you should ask yourself when thinking about how best to present results? 

• Who are my audiences, and how might they best understand the data findings? 

• What are the most important findings, and how can they be highlighted objectively? 

• How can I ensure quality control in presentations? 

• What different media could I use to present the results in the most compelling ways? 

• How can I get feedback to make sure that both the analysis and the presentation are relevant and 
useful? 
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4. Fostering Communication 

You may think the puzzle is solved when you have created a good data set, 
analyzed it in numerous ways to glean the important results, and developed a 
report format or formats that present data results in compelling ways. 

But the most important part of the puzzle hasn't yet been touched. To be 
transformative in an organization, fact-based decision making can't simply 
involve numbers on paper. The results have to be broadly disseminated and openly discussed, and 
adjustments to practice must be made to try and improve outcomes for children, youth, and families. 

One of the most important things that consistent reporting and monitoring can do is help to standardize 
practice across an agency or system of care. That can happen only if differences in performance are 
raised and considered collectively, and practice or policy is adjusted accordingly. Sometimes significant 
differences in behavior are not visible without the data. Sometimes indicators and summary statistics 
often hide variation that needs to be better understood by disaggregating the numbers. By discussing 
the data and the meaning of the findings, the implications can be addressed. 

The events described in the box on the next page could have gone a very different way if staff had been 
discouraged from the group evaluation of results, if offices with the highest rates of out-of-home 
placement had been penalized for those results instead of encouraged to reexamine their practice, or if 
offices with great results were proprietary and secretive about how they did it and what they were 
doing. 

The moral of the story? A well-crafted communication strategy is as important to change as any other 
single factor. A well-crafted communication strategy includes the following: 

Broad dissemination of data to participants at aii ieveis of the system of care, inciuding a wide range 
of stakehoiders and parents and chiidren. A good example of broad communication is the Virginia 
Performs: Measuring What Matters to Virginians website (http://vaperforms.virginia.gov). The website 
tracks benchmark indictors in Virginia for economy, education, health and family, public safety, natural 
resources, transportation, and government and citizens. Virginia's Scorecard compares the state's 
statistics with those of other states nationally, illustrates trends through easy-to-understand graphs, and 
provides regional demographics across the state. 

Another example is the Kids' Well-Being Indicators Clearinghouse (http://www.nyskwic.org), which 
provides New York State indicator data on six major life areas (economic security, physical and 
emotional health, education, civic engagement, family, and community), by county and narratives 
defining each of the indicators and significance for the major life area. The results illustrate differences 
between counties, but further analysis of the data is needed to understand what is happening within the 
county. 

For those who are using data internally, the periodic circulation of results on a schedule that can be 
adhered to, with consistency in measures, methodology, and explanations of both good and bad results, 
will be critical. 
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Revealing Underlying Value Conflicts 

A state child welfare agency had widely diverse regions: some rural, some urban, some highly stable in 
population and economy, and some very transient. For at least a decade, the agency's mission statement 
included a goal to "keep the maximum number of children safely in their own homes or in permanent family¬ 
like settings in their own communities." Everyone in every office saw this statement in their reception area 
daily; many also had it prominently in their own cubicle and on the agency letterhead. But three different 
offices in the organization interpreted its meaning differently. 

The agency created a data dashboard with a special emphasis on family strengthening and reunification and 
encouraged staff within and between offices to explore the meaning of the results. 

Office A had the highest rate of out-of-home placement in the state. Office B had the highest rate of 
permanency planning, with family reunification as the goal. Office C had by far the lowest rate of out-of-home 
placement—but also the highest incidence of family sexual abuse in the state. 

The three office administrators—noting these dramatic differences—decided to collaborate to see whether 
they could ensure consistent practice across the three areas to achieve results closer to those of Office B. Every 
month when the data dashboard was released, staff from the three offices had conference calls to review their 
comparative results and consider how they were operationalizing the agency's values. It became clear to all 
that Office A staff needed to be exposed to successful family reunification programs; they weren't yet 
convinced that it really could work. Office C staff needed more support on safety issues so that they could 
anticipate placements at high risk of sexual abuse and intervene to help eliminate the risk or find an alternative 
permanent result. 

Over time, the three offices did achieve similar results, and staff became skilled at both family strengthening 
and reunification. Within a year, all three offices started posting their results monthly right under the mission 
statement. The data-driven effort had, in fact, become transformative. 


Formal and informal opportunities to discuss and brainstorm results without fear of punishment or 
discipline. As soon as dashboards or other monitoring or tracking tools are used to penalize individual 
performance, the incentive for practice reform will be undermined. Systems of care that are successful 
create a culture that uses quality improvement indicators for positive rather than punitive purposes. 
They noted the importance of using these data to self-inform, self-correct, self-improve, and 
self-monitor. The culture emphasizes the use of data to inform staff on how the system of care can be 
adjusted and how to strive for excellence when serving youth (Kukla-Acevedo, et. al. 2008). 

Expect a lot of pushback and protests. Although people will eventually get to a place where they can 
acknowledge they aren't doing so well, there is great fear of data being used to point out flaws rather 
than improve services. There are often four phases of organizational response to data: 

• Phase I - "The data are wrong." 

• Phase II - "The data are right, but it's not a problem." 

• Phase III - "The data are right; it's a problem, but it's not my problem." 

• Phase IV - "The data are right; it's a problem, and it's my problem." 

(Improvement Tip: Take the Journey to "Jiseki." Institute for Healthcare Improvement, University of MN. 
http://www.ihi.org\knowledge\Pages\lmprovementStories\lmprovementTipTaketheJourneytoJiseki.asp 
x). 
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Discuss measures in context. If you looked solely at the 
reduced number of adoptions finalized in a recent year, you 
might think that permanency efforts had declined when 
compared to a higher number of adoption finalizations 3 
years ago. However, if you examine the picture from a 
wider lens, you might find that there had been a significant 
drop in the rate of children entering placement as well as an 
overall decrease in the number of children waiting for 
adoptive homes. In this situation, the lower number of 
adoption finalization is a sign of improvement, not a 
concern. 

Staff and others need "safe havens" to put their 
assumptions on the tabie and honestiy taik through 
resuits. This needs to be a part of the/r work—not simply 
something they can do on coffee breaks or after hours. It has to be built into the workweek or in some 
way structured into the day-to-day demands of practice. As Jacquelyn McCroskey, data committee chair, 
Los Angeles Children's Planning Council, stated, '"Fast is slow; slow is fast.' Going slow at first to build 
relationships, and ensure that your data and decision-making are credible will enable you to move more 
quickly later" (Los Angeles Children's Planning Council Uses the Power of Outcomes and Indicators from 
Planning to Budgets. Downloaded on March 28, 2012, from Results-Based Accountability™ and 
Outcomes-Based Accountability™: 

http://www.raguide.org/RA/los_angelesJ_Childrens_Planning_Council.htm). 


Communicating Through Data 
Snapshots 

The New York City (NYC) Administration 
for Children provides online community 
data snapshots. These community 
profiles provide detailed information, 
broken down by NYC community 
district, in the following areas: 
community demographic indicators, 
abuse/neglect victimization rates, foster 
care census and placement information, 
preventive services enrollment and 
referral figures, and child care and Head 
Start enrollment statistics. 


Discussions need to focus on problem solving. Are we really placing too many children out of the home? 
What was the basis for this decision in each case? Are we not 
uniformed enough both in safety assessments and in 
functional assessments of children's emotional and intellectual 
abilities? What could we do to break this pattern if indeed we 
are serving families differently than do colleagues in other 
parts of our system of care or elsewhere? 

Making time to step back and reflect on the data and 
collaboratively problem solve provides an opportunity to 
improve practices and ultimately improve outcomes for 
children and families. 

dear, weii-designed reports and presentations expiain why 
you have chosen the measures you have chosen and the 
impiicationsfor the system and the community if you succeed 
in improving resuits on your key goais. Stakeholders at all 
levels of the organization and across the jurisdiction need to 
hear success stories—about the families and children who 
have benefited from new interventions and approaches that 
resulted from being accountable to the information the data 
provided. Even small incremental steps that demonstrate 
changes in the right direction are worth sharing. 


Virginia: 

Using Data to Transform Practice 

• Use real stories to go with the 
numbers. Every monthly meeting 
started out with a family or kid's story 
(experience/inspiration) and followed 
up with data to illustrate the story. 

• Celebrate successes. We never 
backed off sharing data at every 
meeting but made sure to celebrate 
when the numbers moved. 

• Provide incentives. Not only were 
there pressures that the data would 
be made public to media, but 
Virginia had a policy change so that 
there were county fiscal incentives 
for reducing residential treatment 
(increase match rate) and 
increasing community-based services 
(e.g., wraparound, diversion) by 
decreasing match rates. 
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Broadly disseminating the results of the findings takes time and commitment. Those with expertise in 
social marketing can help get the information out to audiences (e.g., legislators, policymakers, and 
media) who can act upon the information in ways that benefit children and families. Instead of 
responding defensively to critics or the media, administrators armed with data can tell a more complete 
story and help key audiences understand the root causes of the challenges faced by children and 
families (Messelt, 2004). 

Summary 

The advice on communications is simple: 

• Disseminate, disseminate, disseminate. Discuss, debate, and decide. 

• Confess sins before others have a chance to hammer you. Toot your horn when you can—but 
humbly. 

• Keep the communication process strength-based—and try to keep media and consumer focus 
there as well. 

• Use results to develop staff—not to discipline them. 

• Encourage everyone to use and share data that inform progress toward outcomes. 

What questions do you need to ask yourself as you develop a communication strategy? 

• Who are my most important internal and external audiences? 

• What am I hoping they will do in response to progress reports and monitoring? 

• What resources do I need to disseminate as broadly as I want? What media will work best? Does 
everybody need everything? 

• Do we have the right skills available on a full-time basis to design and execute a communication 
strategy? 

• How can I best justify the dedication of these resources? 

• What role should children and families play in the communication process, if any? 

• How can I ensure quality control and obtain regular feedback about the data and the 
communication process? 
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5. The Complete Puzzle 

You may be asking yourself whether all this is really worth it. Focusing 
on data about the children and their families is an important step in 
improving practice and outcomes. By breaking down the data values, 
comparing different kinds of data, and looking for strengths, specific 
problems, and potential solutions can be identified. As the wise 
caseworker noted, "Data doesn't do the work; it just identifies the 
work that needs to be done." 

More transparent internal communication, more effective 
collaboration among and between service systems, more accessible technology, and a much greater role 
for children and families are all factors that truly make data-driven, decision-making initiatives 
successful. 

Introducing data-driven decision making can be transformative—that is, providing information to 
transform systems and the delivery of services for children, youth, and their families. However, to be 
transformative, a data-driven, decision-making culture needs to: 

• Begin by asking the right 
questions, clarifying what 
you want to achieve, and 
collecting the data to 
answer the question. 


• Analyze the data on the 
basis of research tools as 
well as shared experiences. 


• Share the data. 

• Use the data to make 
informed decisions and for 
continuous quality 
improvement by ensuring 
there is a process for 
determining how best to 
resolve identified 
problems. 

• Track accomplishments 
toward intended goals. 

• Reward and provide incentives for efforts to improve practice to achieve outcomes. 

Figure 7 displays Erie County's poster for their change results. It shows a significant increase in total 
dollars and percentage allocated to community services from 2005 to 2009 while saving more than $2 
million in county tax. They reduced the cost of congregate care by more than $10 million annually and 


Figure 7. Building a Community System of Care Through Reinvestment 
of Savings From Residential Downsizing 

System of Care: Change in Total and County Expenditures by Year 
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Adapted from "Critical Data Dashboard," by Child and Adolescent Treatment Services, Erie 
County Department of Mental Health, New York, March 2010. 




30 

































Putting the Pieces Together: Guidebook for Fact-Based Decision Making to Improve Outcomes for Children and 
Families 


reinvested the local share of those funds in the system of care. Although budget savings were important, 
the real goal of transformation was improved outcomes for children and families. 

Among the more impressive results to come out of Virginia's transformation process were $100 million 
in budget savings in the first 2 years alone, mostly from reduced reliance on congregate care. State 
spending on congregate care was down 21.5%, whereas state spending on community-based services 
was up 26.1% (Figure 8) (Walters, 2011). 

By changing attitudes about 
data, encouraging 
cooperation and 
collaboration among agency 
and community partners, 
and harnessing technology 
to monitor change, fact- 
based decision making can 
be a driving force for system 
reform and improving 
outcomes for children and 
families (Webster, Needell, 

& Wildfire, 2002). One of 
the ironies of doing child 
welfare work—or, indeed, 
any human services—is that 
while every case is unique, 
consistent outcomes can 
only be achieved with a 
deep, cumulative 

understanding of the broadest possible array of cases and ongoing collective and objective analyses of 
what works best in which circumstances. 

Leaders, managers, supervisors, and direct service staff across human services must marry the passion 
that drew them to the work with data literacy and analytic curiosity so that a sustainable body of 
knowledge can be accumulated and shared to improve the quality and delivery of services and achieve 
consistently successful outcomes. Further, demonstrating the value or gaps in services through data is 
critical to secure adequate resources and funding. 

That's why data matters so much. This guide can serve as an introduction for people who haven't given 
it much thought, a reminder for people who know they should pay attention but lost their focus in the 
face of bureaucracies' frustrations, and an inspiration for people who can't resist digging deeper to 
improve practice and achieve better outcomes for the children and families serve. 
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Appendix A: Sources for Data and Examples of Data Indicators 

Centers for Disease Control and Prevention - Youth Risk Behaviors 
(http://www.cdc.gov/HealthyYouth/yrbs/index.htm) 

Introduction to Cross System Data Sources in Child Welfare, Alcohol and Other Drug Services, and Courts. 
Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration, 
HHS. (http://www.ncsacw.samhsa.gov/files/DataPrimer_508.pdf) 

Child Trends Databank - The one-stop source for the latest national trends and research on more than 
100 key indicators of child and youth well-being (http://www.childtrendsdatabank.org/) 

Childstats.gov - Offers easy access to statistics and reports on children and families, including: family 
and social environment, economic circumstances, health care, physical environment and safety, 
behavior, education, and health (http://childstats.gov/) 

Compendium of NationalJuvenile Justice Data Sets (www.ojjdp.gov/ojstatbb/compedium/index.html) 

Directory of Health and Human Services Data Resources (http://aspe.hhs.gov/datacncl/datadir) 

Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) - New and 
current research findings on child development (http://www.nichd.nih.gov/) 

Health Indicators Warehouse provides indicators by topic, state, or initiative. 
(http://www.healthindicators.gov) 

KIDS COUNT- State- and city-level data for multiple measures of child well-being 
(http://datacenter.kidscount.org/) 

The Montgomery County Collaboration Councii for Children, Youth, and Families — Monitors 30 key 
indicators of child well-being that, taken together, convey the current reality of child and family well¬ 
being in Montgomery County, Maryland. 

(http://www.collaborationcouncil.org/measuring_success_keyJndicators.html) 

NYS Early Childhood Comprehensive Systems Planning Initiative - A framework of priority cross-sector 
goals and outcomes on emotional and physical health, education, family, community, and citizenship, 
where each life area has a set of goals and objectives that represent expectations about the future 
(www.nyskwic.org) 

Nationai Center for Education Statistics (http://nces.edu.gov/ccd) 

Nationai Survey of Chiidren's Heaith - Data collected on children's health that allows for comparisons 
between states (http://childhealthdata.org/learn/NSCH) 

Oregon Benchmarks (OBMs) -The benchmarks measure progress toward Oregon's strategic vision and 
are organized into seven categories: economy, education, civic engagement, social support, public 
safety, community development, and environment (http://benchmarks.oregon.g 0 v/#STEP_ONE) 
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The Urban Institute - The report State Mental Health Systems for Children: A Review of the Literature 
and Available Data Sources includes appendices listing children's mental health data sources and 
findings (http://www.urban.org/UploadedPDF/412207-state-mental.pdf) 

U.S. Census Bureau - The 2012 Statistical Abstract: State Abstracts 
(http://www.census.gov/compendia/statab/st_abstracts.html) 

Virginia Performs - An initiative of the Council on Virginia's Future, a performance leadership and 
accountability system within state government (http://vaperforms.virginia.gov) 
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KIDS COUNT Project and the Social Work Program, University of South Dakota. 

Retrieved from http://www.usd.edu/business/south-dakota-kids-count/upload/DataPrimer08.pdf 

The guide lays out step-by-step instructions for using and presenting data. Topics covered are identifying 
the purpose of data projects, matching data needed to the salient purposes of disparate projects, 
mastering organizational tasks, identifying data sources; applying calculations, and presenting data. This 
guide also reviews the differences between primary and secondary data, the data assembly process, and 
how to import secondary source data into databases. It provides instruction on how to set up 
spreadsheets and online databases, as well as how to use mathematics and statistical procedures to 
manipulate data. This guide also assists learners with the persuasive presentation of statistical data 
using charts, bar graphs, line graphs, and even geographic information systems (GIS). 

Carrillo, T. E., Packard, T., & Clapp, J. D. (2003). Nothing in-nothing out: Barriers to the use of 
performance data in social service programs. Administration in Sociai Work, 27(4), 61-76. 

Retrieved from http://centerforaod.sdsu.edu/pdf/29_nothingJn_nothing_out.pdf 

The document presents data from a multisite evaluation of a state family support program. The authors 
discuss their observations of the participating programs' use of information systems to assist in the data 
collection process. The results of the evaluation show that programs did not consistently use 
information systems even when supported by technical support and training. The authors name this 
problematic phenomena as "nothing in, nothing out," and discuss its implications for data-based 
program planning and management. 

The Casey Outcomes and Decision-Making Project at American Humane Association. (1998). Assessing 
outcomes in chiid weifare services: Principies, concepts, and a framework of core indicators. 
Englewood, CO: American Humane Association. 

Retrieved from http://www.aecf.org/upload/publicationfiles/assessing%20framework.pdf 

The publication describes principle-driven, outcome-focused approaches to decision making in child 
welfare services and applications of these concepts in managed care environments. The purpose of this 
document is to enhance the understanding of outcomes and outcome components and to provide 
essential information about the philosophical principles, concepts, and frameworks for core indicators 
involved in outcome-based child welfare services. Tools included are a matrix of core children's services 
outcome indicators, a framework for organizing various outcome domains, and indicators that are 
appropriate for different communities and systems of care. 

Lampkin, L., & Hatry, H. (2003). Key steps in outcomes management (series on outcome management 
for nonprofit organizations). Washington, DC: Urban Institute. 

Retrieved from http://www.urban.org/UploadedPDF/310776_KeySteps.pdf 

The guidebook is intended to help nonprofits seeking to improve efforts to focus on the outcomes of 
service provision and includes several practical, concise sample schedules and exhibits, including key 
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steps to implement outcome management, suggestions for potential members of outcome 
management working groups, a sample outcome management system development schedule, sample 
agendas for working group meetings, focus group steps; outcome-sequence charts, suggestions for 
linking outcomes to outcome indicators to data sources, and sample report formats. The guide discusses 
four key areas of outcome management: setting up the organizational process, deciding what and how 
to measure, analyzing the data, and using the results to improve services. 

National Child Welfare Resource Center for Organizational Improvement. (2002). A framework for 
quality assurance in child welfare. Portland, ME: Edmund S. Muskie School of Public Service. 

Retrieved from http://muskie.usm.maine.edu/helpkids/rcpdfs/QA.pdf 

Designed for child welfare administrators and senior managers to assess their agency's success in 
working with children and families, this guide discusses the importance of quality assurance, how to 
adopt outcomes and standards, how to gather and analyze data and information, and how to use data 
analysis to make improvements. 

Powell, E., & Renner, M. (2003). Analyzing qualitative data. Madison, Wl: Cooperative Extension 
Publications, University of Wisconsin. 

Retrieved from http://learningstore.uwex.edu/Assets/pdfs/G3658-12.pdf 

The short but informative guide explores systematic ways of analyzing narrative data. The authors 
provide definitions of commonly used terms and give examples of sources of text or narrative data. The 
guide deconstructs the data analysis process, laying out uncomplicated descriptions of each step in the 
process: getting to know the data, focusing the analysis, categorizing information, identifying patterns 
and connections within data categories, and synthesizing and interpreting data. The guide also includes 
practical recommendations on ways to increase the integrity of the data reporting process through 
checking data, identifying data sources, tracking and justifying the choices made, choosing data, 
addressing limitations and alternatives, and more. 

Results-Oriented Management in Child Welfare—A Web-Based Training Site. Kansas University. 

Retrieved from https://rom.socwel.ku.edu/ROMTraining/ 

The site has a series of free web-based training modules designed to show participants how to use 
outcome data to inform improvement efforts, interpret outcome performance data, and establish a 
results-oriented workplace culture. 

Walker, K., Farley, C., & Polin, M. (2012). Using Data in Multi-Agency Collaborations: Guiding 
Performance to Ensure Accountability and Improve Programs. Public/Private Ventures & Child Trends. 

Retrieved from http://www.childtrends.org/Files/Child_Trends-2012_02_23_FR_UsingData.pdf 

The guide provides examples of multiagency collaborations that have employed electronic performance 
management systems to inform sustainable programming and ensure accountability. Topics of 
discussion include tactics for getting started with cross-agency data collection collaboration, how to use 
data to improve outcomes, and program sustainability. 
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